Improve resilience to delayed shared file system visibility - #7489

Surgical fix for #7247 — stochastic "null/truncated/empty file" failures on K8s + CephFS.

Analysis

The two logs attached to the issue show two symptoms of one mechanism.

nextflow.2.log (25.10.5, nf-k8s 1.2.2) — SAMTOOLS_SORT (WT_REP1):

11:25:47.516  [K8s] Cannot read exitstatus for task: `...` | For input string: ""
11:25:47.549  ProcessFailedException: ... terminated for an unknown reason

The pod reported exit code 0, but 25.10.5 evaluated task.exitStatus = k8sExitCode ?: readExitFile(), and 0 is falsy in Groovy — so a successful task always fell back to reading .exitcode, which came back empty. This particular symptom is already fixed on master by 454a2ae (#6484, v25.11.0-edge), which is why the reporter only observes it on v25.

nextflow.3.log (26.04.1, nf-k8s 1.5.2) — STAR_GENOMEGENERATE:

12:58:47.451  Task completed > ... exit: 0
12:58:47.492  MissingValueException: Missing environment variable: nxf_out_eval_92

41 ms after the API reported the pod terminated, .command.env parsed cleanly but was missing its last entry. That file is written by a sequence of echo … >> .command.env appends, so a short read loses the trailing keys.

Root cause. Nextflow decides a K8s task is finished from an out-of-band signal (the pod terminated state), then immediately reads the work directory over a different mount of the shared filesystem. Nothing forces the task's writes to be visible on the driver first. Two code-level facts make it worse:

1. FileHelper.isPathSharedFS() only matched nfs and lustre, so on CephFS workDirIsSharedFS was false and every existing lag mitigation was disabled.
2. on_exit wrote .exitcode before running {{sync_cmd}}, so even NXF_ENABLE_FS_SYNC=true gave no write-ordering guarantee.

Ironically #6484 also removed the accidental read barrier: in v25 the driver at least touched the filesystem before finalizing. In v26 it goes straight from API status to .command.env, so the symptom moved rather than disappeared.

Changes

1. FileHelper — recognise ceph, beegfs and gpfs as shared filesystems. Classification extracted into isSharedFsType(String) so it can be tested independently of the filesystem the tests run on.

2. command-run.txt — move {{sync_cmd}} ahead of the .exitcode write, so the completion marker is only published after everything else the task wrote has been flushed. sync_cmd remains opt-in via NXF_ENABLE_FS_SYNC.

3. K8sTaskHandler — defer task completion until .exitcode is visible and non-empty, up to executor.exitReadTimeout (default 270 s), mirroring GridTaskHandler.readExitStatus(). Because the exit file is the last file the wrapper writes, it acts as a barrier for the whole work directory — covering .command.env, .command.out and the output globs, not just the exit status.

   The wait is skipped when the work dir is not a shared FS, when Fusion is enabled, and when the API reports a non-zero exit code: a task killed by the system (e.g. OOMKilled) never writes the exit file, so waiting would only delay the error report by 270 s.

Relation to #7281

#7281 addresses the same issue across 25 files (~1000 lines). This is the minimal subset. It deliberately omits the per-file retry layer (TaskFileRetry around .command.env / .command.out): a retry can detect a missing file, but a truncated .command.out or short output file is indistinguishable from a correct one. Only a barrier on the last-written marker generalises to all downstream reads.

It also keeps sync_cmd opt-in rather than defaulting it on for shared filesystems — sync on a busy HPC node flushes every client's dirty data, and change 3 carries the reliability on its own.

Trade-offs

• A genuinely lost .exitcode on a shared FS now costs up to exitReadTimeout per task instead of failing fast. Bounded, and the non-zero-exit-code carve-out keeps the common failure paths fast.
• Change 2 leaves .exitcode itself unflushed. That is what the bounded wait in change 3 (and the pre-existing grid polling) is for.

Tests

Scoped to the three modified classes:

• FileHelperTest — should detect shared file system type
• BashWrapperBuilderTest — should flush the file system before writing the exit file, plus the two rendered-wrapper golden files
• K8sTaskHandlerTest — 4 new specs covering the barrier, the timeout, the non-shared-FS bypass and the failed-pod bypass

Locally: 245 tests, 0 failures across the three classes. Full suite left to CI.

Neither I nor CI can reproduce the race itself — only @BioWilko's cluster can confirm the end-to-end fix.